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Description 

[0001 ] The present Invention relates to a tone and pic- 
ture generator device which can generate tones and vis- 
ually display a performance scene of the generated 
tones in three-dimensional pictures. 
[0002] In the field of electronic musical instruments 
and the like, it has been conventional to execute an au- 
tomatic performance, such as an automatic rhythm or 
bass-chord performance, in accordance with a desired 
automatic performance pattern. Specifically, for chord- 
backing and bass parts, chord-backing and bass tones 
are automatically performed in accordance with prede- 
termined automatic performance patterns on the basis 
of chords that are sequentially designated by a human 
player as a music piece progresses. For performance 
of a drum part, on the other hand, normal and variation 
patterns are arranged in advance so that an automatic 
performance can be executed by selecting any of these 
patterns (styles). The number of the arranged variation 
pattern is not always one, and in some cases two or 
more variation patterns are arranged previously. Gen- 
erally, each of these performance patterns has a length 
or duration corresponding to one to several measures, 
and a successive automatic rhythm performance is car- 
ried out by repeating any of these previously-arranged 
performance patterns. 

[0003] With such a conventional approach, the per- 
formance tends to become monotonous because it is 
based on repetition of the same pattern. To avoid the 
undesired monotonousness, it has also been customary 
in the art to previously arrange sub-patterns, such as 
those called "fill-in", "break" and "ad-lib", so that a per- 
formance based on any of these sub-patterns may be 
inserted temporarily in response to an instruction given 
by a human operator or player via predetermined 
switches or the like and then restored to a main pattern 
performance. The main pattern and sub-patterns are 
stored in a database, from which they are retrieved for 
reproduction in response to player's operation. 
[0004] Fig. 11 is a block diagram showing exemplary 
transitions of various performance patterns (styies) in 
an automatic performance. The performance patterns 
in the illustrated example include first and second main 
patterns A and B (i.e., a normal pattern and a variation 
pattern), and two sets of first and second fill-in patterns 
corresponding to the main patterns A and B; that is, the 
two sets are a "A-»A" fill-in pattern ("FILL AA" pattern) 
to be inserted during performance of the first main pat- 
tern A and a "A-»B" fill-in pattern ("FILL AB" pattern) to 
be inserted for transition from the first main pattern A to 
the second main pattern B, and a "B->B" fill-in pattern 
("FILL BB" pattern) to be inserted during performance 
of the second main pattern B and a "B-»A" fill-in pattern 
("FILL BA" pattern) to be inserted for transition from the 
second main pattern B to the first main pattern A. The 
performance patterns of Fig. 1 1 further include two pairs 
of intro patterns ("INTRO A" and "INTRO B") and ending 



patterns ("ENDING A" and "ENDING B") corresponding 
to the two main patterns A and B. 
[0005] Although not specifically shown in Fig. 11, 
there are provided two fill-in pattern selecting switches 
5 ("FILL A" and "FILL B" switches) that are activated when 
one of the patterns (styles) is to be shifted to or replaced 
by another, two switches ("ENDING A" and "ENDING 
B" switches) for selecting a desired ending pattern, and 
two other switches ("INTRO A" and "INTRO B" switches) 
10 for selecting a desired one of the intro patterns. 

[0006] For example, once the "INTRO A" switch is ac- 
tivated, the "INTRO A" pattern is first performed and 
then a performance of the first main pattern A is initiated 
upon termination of the "INTRO A" pattern performance. 
*s |f the "FILL A" switch is depressed during the course of 
the performance of the first main pattern A, the "FILL 
AA" pattern is inserted and then the performance of the 
first main pattern A is resumed. Then, when the "FILL 
B" switch is depressed, the "FILL AB" pattern is inserted 
20 and then the main pattern B is performed. Once the 
"ENDING A" switch is depressed, the "ENDING A" pat- 
tern is performed to stop the performance of the entire 
music piece in question. 

[0007] Similarly, once the "INTRO B" switch is activat- 
es ed, the "INTRO B" pattern is first performed and then a 
performance of the second main pattern B is initiated 
upon termination of the "INTRO B" pattern performance. 
If the "FILL A" switch is depressed during the course of 
the performance of the second main pattern B, the "FILL 
30 BA" pattern is inserted and then the first main pattern A 
is performed. Then, when the "FILL B" switch is de- 
pressed, the "FILL BB" pattern is inserted and then the 
second main pattern B is resumed. Once the "ENDING 
B" switch is depressed, the "ENDING B" pattern is per- 
35 formed to stop the performance of the entire music piece 
in question. 

[0008] In this way, a fill-in pattern is selected, depend- 
ing on the performance state when any one of the 
switches is depressed; corresponding to the currently- 
40 performed main pattern and destination (shifted-to or re- 
placing) main pattern, and the thus-selected fill-in pat- 
tern is inserted. Such fill-in pattern insertion can effec- 
tively avoid unwanted monotonous of the music piece 
performance. 

45 [0009] While Fig. 11 shows a case where two main 
patterns A and B are used, the number of the main pat- 
terns is of course not so limited and may be more than 
two. Further, there have been known various other man- 
ners of pattern variations and transitions than the above- 

50 mentioned; for example, the fill-in pattern insertion may 
be applied only to a selected musical instrument of a 
single performance part. 

[0010] Among other known types of automatic per- 
formance devices than the above-discussed device is 
55 one which prestores, as SMF (Standard MIDI File)-for- 
mat performance information, pitch, sounding-start and 
muffling-start timing, etc. Of each note contained in a 
desired music piece and generates tones by sequential- 
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ly reading out the prestored pieces of the performance 
information (composition data). In this know automatic 
performance device, a human player only has to operate 
performance-start and performance-stop switches. 
[0011] However, the conventionally-known electronic 5 
musical instruments, having functions to execute an au- 
tomatic accompaniment and automatic performance, 
could not carry out a visual interaction with the users or 
players although they could provide an interaction by 
sound (aural interaction). 10 
[0012] Some of the known electronic musical instru- 
ments are provided with a display section for visually 
showing a title of an automatically-performed or auto- 
matically-accompanied music piece and/or changing 
measures and tempo during the performance. Also 1$ 
known is a technique by which each key to be next de- 
pressed by the player is visually indicated on the display 
section. 

[001 3] For an example of the prior art see US-A-5 005 
459. 20 
[001 4] It is therefore an object of the present invention 
to provide a tone and picture generator device which can 
display performance motions, corresponding to a per- 
formance style, in synchronism with a music perform- 
ance, to thereby allow a player to perform while viewing 25 
and enjoying performance of various musical instru- 
ments. 

[0015] In order to accomplish the above-mentioned 
object, the present invention provides a tone and picture 
generator device as defined in claims 1 and 1 0, a meth- 30 
od as defined in claim 7 and a machine readable record- 
ing medium as defined in claim 9. 
[0016] By virtue of the database storing the motion 
components, common or same motion components can 
be used for a plurality of different patterns or music piec- 35 
es, and any necessary components can be additionally 
stored in the database whenever necessary. As a con- 
sequence, various 3-D animated pictures can be gen- 
erated with increased efficiency. The use of such 3-D 
animated picture data allows the users to enjoy more *o 
real, stereoscopic animated pictures. 
[0017] Further, according to the present invention, 
each of the motion components includes not only the 
motion information representative of a trajectory of per- 
formance motions of a subdivided performance pattern 45 
but also a sounded point marker indicative of each tone- 
generation timing in the motion information. Thus, com- 
mon motion components can be used for different per- 
formance tempos, which thereby permits a significant 
reduction in the size of the database. Further, using the so 
sounded point marker for synchronization with the tone 
generator section, the tone and picture can be synchro- 
nized with each other with high accuracy. 
[0018] In addition, the present invention allows a hu- 
man operator or player to change a "character" playing 55 
in the performance scene to be displayed and viewpoint 
of the 3-D animated picture data, so that the human op- 
erator can enjoy a variety of 3-D animated pictures and 



also can cause a model performance to be displayed on 
a magnified scale. 

[0019] The tone and picture generator device of the 
present invention may further comprise a section for 
modifying the motion information in response to a 
change in the playing (player-representing) character 
and/or viewpoint. With this modifying section, common 
motion information carvbe used for different player-rep- 
resenting characters and viewpoints, which can even 
further reduce the size of the database. 
[0020] For better understanding of the object and oth- 
er features of the present invention, its preferred em- 
bodiments will be described in greater detail hereinbe- 
low with reference to the accompanying drawings, in 
which: 

Fig. 1 is a block diagram showing an exemplary or- 
ganization of a tone and picture generator device in 
accordance with an embodiment of the present in- 
vention; 

Fig. 2 is a diagram showing an exemplary outward 
appearance of the tone and picture generator de- 
vice shown in Fig. 1 ; 

Fig. 3 is a diagram explanatory of a motion compo- 
nent database employed in the tone and picture 
generator device of Fig. 1 ; 

Fig. 4 is a flowchart illustrating an exemplary oper- 
ational sequence of a motion component creation 
process executed in the generator device; 
Figs. 5A and 5B are schematic diagrams explana- 
tory of the motion component creation process; 
Fig. 6 is a flow chart illustrating operational se- 
quences of a picture generation/display process 
and a tone generation process to be executed dur- 
ing an automatic accompaniment in the generator 
device; 

Figs. 7A to 70 are diagrams explanatory of an ex- 
ample of a basic motion information creation proc- 
ess executed in the generator devjce; 
Fig. 8 is a diagram explanatory of a coordinates 
modification process executed in the generator de- 
vice;. 

Fig. 9 is a flow chart illustrating operational se- 
quences of a picture generation/display process 
and a tone generation process executed during an 
automatic performance in the generator device; 
Fig. 10 is a diagram showing another example of 
the outward appearance of the tone and picture 
generator device; and 

Fig. 11 is a block diagram showing an exemplary 
transition of performance patterns occurring during 
an automatic accompaniment. 

[0021] Fig. 1 is a block diagram showing an exempla- 
ry organization of a tone and picture generator device 
in accordance with an embodiment of the present inven- 
tion. In Fig. 1 , the tone and picture generator device in- 
cludes a central processor unit (CPU) 1 for controlling 
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various operations to be performed in the entire device, 
a program storage 2 for storing a control program to con- 
trol this tone and pictu re generator device, and a storage 
unit 3, such as a ROM and RAM, which contains a style 
database storing various automatic performance pat- 
terns such as rhythm patterns and automatic bass- 
chord patterns, motion-component and scene-compo- 
nent databases for generation of a three-dimensional 
(hereinafter "3-D") picture indicative of a current scene 
or situation of a performance and which is also used for 
storing various other data and as working areas for the 
CPU. Moreover, the tone and picture generator device 
includes a keyboard/operation switch group provided on 
an operation panel, which includes a keyboard and var- 
ious operators such as button switches to be described 
later. Reference numeral 5 denotes a tone generator 
section that generates signals of scale tones and rhythm 
tones for a plurality of channels using any one of the 
known tone generation schemes such as the waveform 
memory scheme, FM scheme, physical model scheme, 
harmonics synthesis scheme, formant synthesis 
scheme and analog synthesizer scheme based on a 
well-known combination of VCO, VCF and VGA. The 
tone generator section is not necessarily limited to a cir- 
cuit based on dedicated hardware; it may be a tone gen- 
erator circuit based on a combination of a DSP and mi- 
croprograms or a combination of a CPU and software 
program. The tone generator section 5 also includes an 
effect processing (effector) section that imparts various 
effects, such as a vibrato and reverberation, to the gen- 
erated tone signals, although not specifically shown 
here. Further, reference numeral 6 denotes a sound sys- 
tem that audibly reproduces or sounds the tone signals 
output from the tone generator section 5. 
[0022] The tone and picture generator device in the 
illustrated embodiment further includes a graphic dis- 
play unit 7, which visually shows operating states of the 
tone and picture generator device as well as operational 
states of the operation switches and which also shows, 
in a 3-D animated picture, a performance scene or sit- 
uation of a selected musical instrument or part. 
[0023] Further, in Fig. 1 , reference numeral 8 denotes 
an external storage device such as a hard disk drive, 
floppy disk drive, CD-ROM drive, MO drive and/or DVD 
drive, and reference numeral 9 denotes a MIDI commu- 
nication interface (l/F) circuit for communication with an 
external MIDI instrument. The tone generator section 5 
is further provided with a video interface circuit 10 for 
displaying the picture indicative of a performance scene 
on an external monitor 11 , and a bus 12 for data transfer 
between the various components mentioned above. 
[0024] Fig. 2 is a diagram showing an exemplary out- 
ward appearance of the tone and picture generator de- 
vice shown in Fig. 1 . In the illustrated example, the op- 
eration switch group 4 includes the keyboard 40; a start 
switch 41 for instructing a start of an automatic perform- 
ance, a stop switch 42 instructing a stop of an automatic 
performance, and a style selection switch set 43 for se- 



lecting performance patterns, such as rhythm, main and 
variation patterns, to be automatically performed. The 
operation switch group 4 also includes an instrument 
change switch set 44 for selecting a musical instrument 

5 or part whose current performance scene is to be visu- 
ally displayed, a player change switch set 45 for select- 
ing a playing (player-representing) character that is to 
be used for displaying the performance scene, a fill-in 
switch set 46 for selecting a musical instrument for 

io which a fill-in pattern performance is to be executed, a 
stage change switch set 47 for selecting a background 
to be used when the performance scene is to be dis- 
played, and a viewpoint change switch set 48 for setting 
a viewpoint when the performance scene is to be dis- 

1* played. Keys labeled "D", "G M , "B" and "K\ on upper 
rows of the above-mentioned instrument change switch 
set 44, player change switch set 45 and fill-in switch set 
46, are provided for selecting a drum part, guitar part, 
bass part and keyboard part, respectively, and keys la- 
20 beled "A", "B", "C" and "D" on lower rows of these switch 
sets are for selecting respective details of the individual 
parts selected via the upper-row keys "D", "G", "B" and 
"K". 

[0025] Further, in the illustrated example of Fig. 2, 

25 performance scenes or situations of a plurality of the 
parts (e.g., three parts consisting of the keyboard, bass 
and drum parts) are being visually displayed on the 
graphic display unit 7 (or on the external monitor 11) in 
a 3-D animated picture. 

30 [0026] Before describing processing for displaying 
such a 3-D animated picture, the motion-component da- 
tabase 20 will be described first. In this motion-compo- 
nent database 20, various performance patterns are 
subdivided for each one of the various musical instru- 

35 ments or parts, and performance motions correspond- 
ing to the subdivided performance patterns are each ac- 
quired as motion capture data, developed in the x-, y- 
and x-axis directions and then stored along with data 
indicative of their respective tone-generation timing (e. 

^0 g., striking points in the case of a drum). The data indic- 
ative of each of the subdivided performance patterns will 
hereinafter be called a "motion component", and the da- 
ta indicative of the respective tone-generation timing will 
be called "sounded point marker" data. 

45 [0027] Fig. 3 is a diagram illustrating motion compo- 
nents for the drum part. As shown, each of the motion 
components stored in the database 20 is made up of 
motion information that is, for one of the subdivided 
drum-part performance patterns corresponding to short 

so phrases A, B, C, D indicative of a motional trajectory 

of a human player during the pattern performance and 
the sounded point marker data corresponding thereto. 
While a single motion component is shown here as be- 
ing composed of the motion information of a set of three 

55 musical instruments, i.e., cymbal, snare drum and bass 
drum, such a motion component is normally created per 
musical instrument in the case of piano, saxophone and 
the like. 
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[0028] Now, a process for generating the motion com- 
ponents will be described more fully with reference to 
the flow chart of Fig. 4. First step S1 0 of this motion com- 
ponent creation process is directed to acquiring, as "mo- 
tion capture data", a motional state of the. player per- 
forming a particular subdivided phrase on a particular 
musical instrument. 

[0029] Fig. 5A is a diagram explanatory of how the 
player's motional state is acquired as the motion capture 
data. As shown, the player is asked to perform the par- 
ticular subdivided phrase with 3-D digitizers attached to 
principal portions of the player's body and, if necessary, 
to the musical instrument as well, and motions of the 
player during the performance are recorded in a se- 
quential manner. The 3-D digitizers employed here may 
be of a known magnetic or optical type. 
[0030] At next step S11 of Fig. 4, trajectories of the 
respective centers of the individual body portions, as 
represented by the thus-acquired motion capture data, 
^ are developed in the x, y z coordinates so as to acquire 
motion information indicative of movements and posi- 
tions of the individual body portions. At that time, time 
data may also be recorded in association with the mo- 
. tion information. 
[0031] Then, the motion creation process moves on 
to step S1 2, where the coordinates of each of the prin- 
cipal body portions at a point where a tone has been 
generated (sounded point) and the elapsed time from 
the start of the performance to the sounded point are 
stored as a sounded point marker in any desired distin- 
guishable form. If the performance is of a phrase shown 
in Fig. 5B, three points labeled "X" in the figure are 
sounded points and the respective elapsed times of 
these sounded points t, t' andt" are stored in distinguish- 
able form. It will be appreciated that these sounded point 
markers may be in any suitable format as long as they 
can properly identify the sounded points from among the 
acquired motion capture data. 

[0032] Following step S12, the process proceeds to 
J step S13, where the data acquired in the above-men- 
tioned manner are associated with the phrase per- 
formed by the player and then stored into the database 
as data in such a format which can appropriately deal 
with any positional changes (e.g., changes in the shape 
" and size of the player and musical instrument) and/or 
time changes (e.g., tempo change) that may take place 
in subsequent reproduction of the acquired data. 
[0033] Note that the above-mentioned motion compo- 
nent data may contain other data, such as those indic- 
ative of respective moving velocity and acceleration of 
the individual body portions, in addition to the x, y and 
z coordinates, time data and sounded point markers. 
[0034] The following paragraphs describe a process 
for generating and visually displaying a 3-D animated 
picture by use of the thus-created motion component 
database 20, in relation to a device equipped with an 
automatic accompaniment function. Fig. 6 is a flow chart 
illustrating operational sequences of a picture genera- 



tion/display process and a tone generation process dur- 
ing automatic accompaniment reproduction; in particu- 
lar, Fig. 6 illustrates an exemplary operational flow for 
reproducing a 3-D animated picture visually showing a 

5 tone of one part and a performance scene correspond- 
ing thereto. If performance scenes of a plurality of parts 
are to be displayed, it is only necessary that the same 
process as shown in Fig. 6 be carried out for each of the 
parts and then the processed results be displayed in a 

w combined format. 

[0035] First, once the player activates any of the 
above-mentioned operation switches to initiate an auto- 
matic accompaniment, a performance style data is se- 
lected from among those stored in the above-mentioned 

15 style database 21 , similarly to the conventionally-known 
automatic accompaniment function. The thus-selected 
performance style data is then delivered to operations 
of steps S21 and S25. 

[0036] Step S25 is directed to the operation similar to 
20 the conventional automatic accompaniment process; 
more specifically, this step generates tone generation 
event data, such as a MIDI key-on event and control 
change, and tone generator controlling parameters (T. 
G. parameters") on the basis of performance informa- 
25 tion included in the selected performance style data. 
The tone generator controlling parameters, etc. gener- 
ated in this manner are then passed to the tone gener- 
ator section 5, which, in turn, generates a corresponding 
tone signal (step S26) to be audibly reproduced through 
30 the sound system 26. 

[0037] At step S21 , the motion components corre- 
sponding to the selected performance style data are se- 
lected from among those stored in the above-mentioned 
motion component database 20, to thereby generate 
35 basic motion information to be described below. Be- 
cause the motion components corresponding to the in- 
dividual performance styles can be known previously, it 
is possible to include, in the selected performance style 
data, such data indicative of the corresponding motion 
40 components. 

[0038] One exemplary process for generating the ba- 
sic motion information will be described in detail below 
with reference to Figs. 7A and 7B, of which Fig. 7A 
shows example phrases corresponding to various mo- 
45 tion components stored in the motion component data- 
base 20. Namely, in this motion component database 
20, there are prestored motion components in associa- 
tion with phrases A, B, C, D, ... shown in Fig. 7A. As- 
suming that a performance pattern corresponding to the 
so selected performance style is the one shown in Fig. 7B, 
the motion components corresponding to the perform- 
ance pattern are read out from the motion component 
database 20. Then, every adjacent motion components 
thus read out from the database 20 are joined together 
55 by causing a trailing end portion of the preceding motion 
component and a leading end portion of the succeeding 
motion component to overlap each other, so as to create 
the basic motion information. Thus, f or the basic pattern 
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of Fig. 7B, the motion components associated with the 
phrases A, B, C, B will be sequentially joined together 
in the mentioned order (A->B-»C-»B). 
[0039] When the player has instructed a variation op- 
eration, such as insertion of a fill-in, for the particular 
musical instrument, the process goes to step S22 of Fig. 
6, where the motion information corresponding to the 
fill-in pattern is caused to overlap or replace the basic 
motion information generated at step S21 . If the style 
pattern to be performed is a variation pattern as shown 
in Fig. 70, i.e., if a fill-in operation is to be effected for 
the cymbal and snare drum in the drum part, the last 
portion of the basic motion information (A->B->C-*B) 
generated at step S21 and the data immediately pre- 
ceding the same are replaced by the data of the motion 
( component D, to thereby provide motion information 
corresponding to the variation pattern. By thus replacing 
part of the motion components with part of another mo- 
tion component, it is possible to properly deal with the 
instructed variation operation. 

[0040] After that, the process of Fig. 6 moves on to 
step S23 in orderto selectively read out, from the scene 
component database 22, the information corresponding 
to displayed-part selection data entered via the above- 
mentioned instrument change switch set 44, playing- 
character selection data entered via the player change 
switch set 45, viewpoint change operation data entered 
via the viewpoint change switch set 48 and stage 
change operation data entered via the stage change 
switch set 47. 

[0041] Step S23 also modifies the coordinates data 
included in the motion component information. Namely, 
step S23 reads out, from the scene component data- 
base 22, the scene components corresponding to the 
part or musical instrument whose performance scene is 
to be displayed, i.e., a player-representing character 
who is performing, selected stage and designated view- 
point (camera position). Note that when an instruction 
is given to simultaneously display a plurality of parts and 
musical instruments, the scene components corre- 
sponding to the positional arrangement of these parts 
or instruments are read out from the database 22. 
[0042] The following paragraphs describe an exam- 
ple of the coordinates modification process, with refer- 
ence to Fig. 8. This example assumes that the musical 
instrument whose performance scene is to be displayed 
is cymbal and the motion information contains a trajec- 
tory of the stick (denoted by "(1)") extending from an in- 
itial position (xO, yO, zO) to a target position (xt, yt, zt) on 
the cymbal. Let's also assume here that the height of 
the cymbal is varied by data such as that of the player- 
representing character or viewpoint selected by the hu- 
man operator, so as to assume a target coordinates po- 
sition (xt\ yt*, zt 1 ). In this case, the above-mentioned mo- 
tion information is modified at step S23 to achieve a tra- 
jectory as denoted by "(2)". When the player- represent- 
ing character has been changed and the initial position 
of the stick has been changed to one denoted by a dot- 



ted line in Fig. 8, the above-mentioned motion informa- 
tion is modified to achieve a trajectory as denoted by " 
(3)". When both the player-representing character and 
the cymbal's height have been changed, the above- 
mentioned motion information is modified to achieve a 
trajectory as denoted by "(4)". 

[0043] In this manner, step S23 sets model positions 
and animated picture corresponding to the model posi- 
tions. 

[0044] Then, the routine goes to step S24, where a 
picture generation (rendering) process is carried out on 
the basis of the information having b^en set at step S23. 
Namely, at this step, the scene is visualized in a video 
form on the basis of the above-mentioned scene infor- 
mation and motion information. More specifically, on the 
basis of the scene information and motion information, 
there are performed coordinates conversion, hidden 
scene erasure, calculation of intersecting points, lines, 
planes and the like, shading, texture mapping, etc. to 
compute the luminance of each pixel and pass it to the 
graphic display unit 7. 

[0045] As previously noted, each of the motion com- 
ponents stored in the motion component database 20 
contains the sounded point marker as well as the coor- 
dinates data along the time axis, so that, in this embod- 
iment, each picture and a corresponding tone can be 
accurately synchronized with each other on the basis of 
the sounded point marker. 

[0046] Namely, on the basis of such sounded point 
markers, it is possible to compute each coordinates po- 
sition and a time length and moving speed from a start 
of reproduction of the corresponding motion information 
to each sounded point. 

[0047] Namely, as previously described in relation to 
Fig. 5B, the time values t, t\ t", at the basic tempo, up 
to each sounded point can be acquired from the motion 
component. Therefore, if a performance tempo has 
been increased by a factor of k from the basic tempo 
with which the motion component was created, it is suf- 
ficient that control be performed forthinning out the mo- 
tion-information reading operations or repeatedly read- 
ing the same motion position so as to make shorter or 
longer the reproduction intervals of the motion informa- 
tion in such a manner that the desired sounded point 
can be reached from the start of reproduction of the mo- 
tion information within only 1/k of the original time (or at 
k times the original speed). Where a moving time or 
speed is prepared for each coordinates position, i.e., 
where information indicative of a time or speed for each 
body portion to move from one coordinates position to 
a next one is contained in the motion information, and if 
such information is representative of time, then the con- 
trol may be executed to modify the time to 1/k of the 
original, or if the information is representative of speed, 
then the control may be executed for modifying the 
speed to k times the original. 

[0048] In this way, it is possible to generate a perform- 
ance picture with an accurate sounded point in accord- 
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ance with the current performance tempo. 
[0049] Further, reliability in synchronizing the tone 
and picture to be generated can be greatly enhanced if 
the picture generating step S24 is arranged to inform 
the tone generator control parameter generating step 
S25 of the arrival at the picture generating process for 
the sounded point. 

[0050] In the above-mentioned manner, the perform- 
ance scene of any selected part can be displayed, in a 
3-D picture, in accurate synchronism with the automatic 
accompaniment data. 

[0051] The following paragraphs describe an exam- 
ple were the principle of the present invention is applied 
to an automatic performance device for reproducing 
composition data of a desired music piece, with refer- 
ence to the flow chart of Fig. 9 (reproduction of an au- 
tomatic performance). Where such a reproductive auto- 
matic performance is to be carried out, performance in- 
formation (composition data) of the music piece to be 
performed is prestored in a composition database 23. 
Once the human operator or player selects a music 
piece to be automatically performed, the composition 
data of the selected music piece are sequentially read 
out at step S30 from the composition database 23, a 
predetermined data length at a time. The read-out data 
are then given to steps S31 and S34, which, similarly to 
steps S25 and S26 of the automatic accompaniment 
process, generates a tone signal based on the read-out 
data and audibly reproduces the tone signal through the 
sound system 6. 

[0052] Steps S31 to S33 are directed to generating a 
3-D animated picture corresponding to the read-out da- 
ta. At step S31 , some of the motion components closest 
. to the predetermined length of the read-out data are se- 
lectively read out. Then, similarly to step S21 above, 
every adjacent motion components thus read out are 
joined together by causing a trailing end portion of the 
preceding motion component and a leading end portion 
of the succeeding motion component to overlap each 
other, so as to create basic motion information. Namely, 
a length of data corresponding to the subdivided phrase 
(hereinafter called a "first segment") is extracted from 
the beginning of the performance data, and the motion 
component corresponding to the phrase closest to the 
extracted first segment is read out from the database 
20. Then, similarly, a second segment is extracted with 
the end of the first segment set at the beginning of the 
second segment, and the motion component corre- 
sponding to the phrase closest to the second segment 
is read out from the motion component database 20 and 
joined to the first read-out motion component. The afore- 
mentioned procedures are repeated to join together 
every subsequent components, to thereby create the 
basic motion information. 

[0053] Whereas the preceding paragraphs have de- 
scribed the case where general-purpose motion com- 
ponents are applied to optionally-selected composition 
data, the motion components may be arranged in stand- 



ardized basic sets (e.g., such that basic tone colors are 
automatically associated by tone color numbers as with 
"GM" basic tone colors), motion component designating 
information, corresponding to the motion components 
5 of. the basis set to be used in the composition data may 
be included in accordance with the progression of the 
music piece. 

[0054] Afterwards, model positions and animated pic- 
ture corresponding thereto are set at step S32 in a sim- 
10 iiar manner to step S23, and then the routine moves on 
to step S33 where, similarly to step S24 above, a 3-D 
animated picture is generated and visually shown on the 
graphic display unit 7. 

[0055] In the above-mentioned manner, a 3-D animat- 
es ed picture representative of the performance scene of 
that music piece can be displayed also in the case of 
the automatic performance. 

[0056] Further, in Fig. 1 0, there is shown another ex- 
ample of the external appearance of the tone and picture 

20 generator device in accordance with the present inven- 
tion, in the illustrated example, various operators are 
disposed to the left and right of the graphic display unit 
7, and a current performance scene of a single part 
(drum part in this case) is being demonstrated in a 3-D 

25 animated picture on the display screen. Here, the oper- 
ator 51 is an automatic-performance start button, the op- 
erator 52 is an automatic-performance stop button, 53 
is a tempo-up button for making the performance tempo 
faster, 54 is a tempo-down button for making the per- 

30 formance tempo slower, 55 is a player selection button 
for selecting a player-representing character to be used 
in showing a current performance scene on the graphic 
display unit 7, and 56 is a musical instrument selection 
button for selecting a particular musical instrument 

35 whose current performance scene is to be shown on the 
graphic display unit 7. Further, the operators 57 and 58 
are buttons for selecting a desired main pattern (main 
style) of an automatic performance; specifically, 57 is a 
main-A button for selecting the A main pattern while 58 

40 is a main-B button for selecting the B main pattern. 59 
is an intro button for selecting an intro pattern, 60 is a 
fill-in button for selecting a fill-in pattern, and 61 is an 
ending button for selecting an ending pattern. Further, 
the operator 62 is a viewpoint moving button for moving 

45 a viewpoint when a three-dimensional performance 
scene is to be shown on the above-mentioned graphic 
display unit 7. 

[0057] In the above-mentioned manner, the current 
performance scene of one or any other number of parts 

50 can be displayed. 

[0058] It should be apparent that the principles of the 
present invention are also applicable to sequencers 
having no keyboard section. Further, whereas the 
present invention has been described above in relation 

55 to an automatic accompaniment or automatic perform- 
ance, it may be used to display a 3-D animated picture 
corresponding to melody-part performance data en- 
tered by manual operation such as key depression. 
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[0059] According to the present invention, the effect 
to be imparted in the tone generator section 5 may be 
changed in accordance with a stage selected via the 
above-mentioned stage change switch set 47. For in- 
stance, the effect may be varied depending on a situa- 5 
tion of the picture to be displayed; that is, if a "concert 
hall stage" is selected, a delay effect may be made 
greater, or if an "outdoor stage" is selected, the delay 
may be made smaller. 

[0060] Furthermore, whereas the present invention w 
has been described in relation to the case where pieces 
of motion information (motion files) are acquired by the 
motion capture scheme, the motion information may be 2 
created by any other schemes than the motion capture 
scheme. 15 
[0061] With the above-mentioned arrangements, the 
present invention can display a 3-D animated pictures 
in synchronism with composition data, so that the hu- 
man operator or player can enjoy visual interaction, 
based on the 3-D animated picture, as well as interac- 20 
tion by sound. 

[0062] Further, by virtue of the database storing mo- 
tion components, common motion components can be 
used for a plurality of different patterns or music pieces, 
and any necessary components can be additionally 25 
stored in the database whenever necessary. As a con- 3. 
sequence, various 3-D animated pictures can be gen- 
erated with increased efficiency. 
[0063] Furthermore, because each of the motion 
components includes sounded point markers in associ- 30 
ation with motion information, common motion compo- 
nents can be used for different performance tempos, 
which permits a significant reduction in the size of the 
database. 4 
[0064] Moreover, with the present invention, the hu- 35 
man operator can select a character, suiting his or her 
preference, from among a plurality of player- represent- 
ing characters. 5 
[0065] In addition, because the human operator is al- 
lowed to change the viewpoint of the displayed picture, 40 
it is possible to observe a model performance scene in 
any desired position, and the thus-shown model per- 
formance scene can be used for teaching purposes as 
well. 6 



Claims 

1 . A tone and picture generator device comprising: 

so 7. 

a tone generator means (5) that generates a 
tone on the, basis of performance information 
(MIDI); 

a storage means (20) that stores a plurality of 
motion components each including motion in- 55 
formation representative of a trajectory ((1 ), (2), 
(3), (4)) of performance motions of a perform- 
ance pattern; and 



a picture generator means that, on the basis of 
the performance information, reads out from 
said storage means one of the motion compo- 
nents corresponding to the performance infor- 
mation and, in synchronism with said perform- 
ance information and on the basis of the motion 
component read out from said storage means, 
generates picture data illustrating a perform- 
ance scene of a selected musical instrument or 
part corresponding to the performance informa- 
tion. 

A tone and picture generator device as recited in 
claim 1 , wherein said motion component represents 
a trajectory of performance motions of a subdivided 
performance pattern for each musical instrument or 
performance part, and wherein said picture gener- 
ator means reads out, from said storage means 
(20), one of the motion components corresponding 
to the performance information and generates ani- 
mated picture data corresponding to the perform- 
ance information on the basis of information that is 
created by sequentially joining together the motion 
components read out from said storage means (20). 

A tone and picture generator device as recited in 
claim 1, wherein said motion component includes 
motion information representative of a trajectory 
((1), (2), (3), (4)) of performance motions of a sub- 
divided performance pattern and a sounded point 
marker indicative of tone-generation timing in the 
motion information. 

A tone and picture generator device as recited in 
claim 1 , wherein said picture data is three-dimen- 
sional animated picture data. 

A tone and picture generator device as recited in 
claim 4 wherein a player- representing character 
and viewpoint illustrated by the three-dimensional 
animated picture data can be changed by a human 
operator. 

A tone and picture generator device as recited in 
claim 5 which further comprises a means that mod- 
ifies the motion information in accordance with a 
change in the player-representing character and 
viewpoint. 

Atone and picture generating method comprising: 

providing performance information (MIDI); 
generating a tone on the basis of the provided 
performance information; 
storing a plurality of motion components each 
including motion information representative of 
a trajectory ((1), (2), (3), (4)) of performance 
motions of a performance pattern; 
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on the basis of the provided performance infor- 
mation, reading out one of the stored motion 
components corresponding to the provided 
performance information; and 
in synchronism with the provided performance 
information and on the basis of the read out mo- 
tion conmponent, generating picture data illus- 
trating a performance scene of a selected mu- 
sical .instrument or part corresponding to the 
performance information. 

8. A tone and picture generating method as recited in 
claim 7, further comprising the step of generating 
animated picture data corresponding to said per- 
formance information (MIDI) on the basis of infor- 
mation that is created by sequentially joining togeth- 
er the motion components read out, wherein said 
plurallity of motion components represent a trajec- 
tory of performance motions of a subdivided per- 
formance pattern for each musical instrument or 
performance part. 

9. A machine-readable recording medium containing 
a group of instructions of a tone and picture gener- 
ating method to be executed by a processor, said 
tone and picture generating method comprising the 
steps of: 

receiving performance information (MIDI); 
generating a tone on the basis of said perform- 
ance information received by the step of receiv- 
ing; 

storing a plurality of motion components each 
including motion information representative of 
a trajectory ((1), (2), (3), (4)) of performance 
motions of a performance pattern ; on the basis 
of the received performance information, read- 
ing out one of the stored motion components 
corresponding to the received performance in- 
formation; and 

in synchronism with the received performance 
information and on the basis of the read out mo- 
tion conmponents, generating picture data il- 
lustrating a performance scene of a selected 
musical instrument or part corresponding to the 
received performance information. 



PatentansprOche 

1. Ton- und Bild-Erzeugungsvorrichtung, die folgen- 
des aufweist: 

Tongeneratormittel (5), die einen Ton auf der 
Grundlage einer Spielinformation (MIDI) erzeu- 
gen; 

Speichermittel (20), die eine Vielzahl von Be- 
wegungskomponenten speichern, von den je- 



de eine Bewegungsinformation enthalt, die fur 
eine Bewegungsbahn ((1), (2), (3), (4)) von 
Spielbewegungen eines Spielmusters repra- 
sentativ ist; und 

s Bildgeneratormittel, die auf der Grundlage der 

Spielinformation aus den Speichermitteln eine 
der Bewegungskomponenten, die der Spielin- 
formation entspricht, auslesen und synchron 
mit der Spielinformation und auf der Grundlage 

w der aus den Speichermitteln ausgelesenen Be- 

wegungskomponente Bilddaten erzeugen, die 
eine Spielszene eines ausgewahlten Musikin- 
struments oder eines Parts, die der Spielinfor- 
mation entsprechen, veranschaulichen. 

15 

2. Ton- und Bild-Erzeugungsvorrichtung nach An- 
spruch 1 , bei der die Bewegungskomponente eine 
Bewegungsbahn von Spielbewegungen eines sub- 
unterteilten Spielmusters fur jedes Musikinstrument 

20 oder jeden Spielpart reprasentiert und bei dem die 
Bildgeneratormittel aus den Speichermitteln (20) ei- 
ne der Bewegungskomponenten, die der Spielinfor- 
mation entspricht, auslesen und animierte Bildda- 
ten, die der Spielinformation entsprechen, auf der 

25 Grundlage einer Information erzeugen, die durch 
sequentielles Aneinanderfugen der aus den Spei- 
chermitteln (20) ausgelesenen Bewegungskompo- 
nenten erstellt wird. 

30 3. Ton- und Bild-Erzeugungsvorrichtung nach An- 
spruch 1, bei der die Bewegungskomponente eine 
Bewegungsinformation, die fur eine Bewegungs- 
bahn ((1), (2), (3), (4)) von Spielbewegungen eines 
subunterteilten Spielmusters reprasentativ ist, und 

35 ein Ertonungspunkt-Markierungszeichen enthalt, 
das fur einen Tonerzeugungszeitpunkt in der Bewe- 
gungsinformation kennzeichnend ist. 

4. Ton- und Bild-Erzeugungsvorrichtung nach An- 
40 spruch 1 , bei der die Bilddaten animierte dreidimen- 

sionale Bilddaten sind. 

5. Ton- und Bild-Erzeugungsvorrichtung nach An- 
spruch 4, bei der ein einen Spieler reprasentieren- 

45 des Zeichen und ein Betrachtungspunkt, die durch 
die animierten dreidimensionalen Bilddaten darge- 
stelltwerden, durch eine Bedienperson geandert 
werden konnen. 

so 6. Ton- und Bild-Erzeugungsvorrichtung nach An- 
spruch 6, die auBerdem Mittel aufweist, die die Be- 
wegungsinformation in Entsprechung zu einer An- 
derung des einen Spieler reprasentierenden Zei- 
chens und des Betrachtungspunkts modifizieren. 

55 

7. Ton- und Bild-Erzeugungsverfahren, das die fol- 
genden Schritte aufweist: 
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8. 



9. 



Bereitstellen einer Spielinformation (MIDI); 
Erzeugen eines Tons auf der Grundlage der be- 
reitgestellten Spielinformation; 
Speichern einer Vielzahl von Bewegungskom- 
ponenten, die jeweils eine Bewegungsinforma- 5 
tion enthalten, die fur eine Bewegungsbahn 
((1), (2), (3), (4)) von Spielbewegungen eines 
Spielmusters reprasentativ ist; 
Auslesen einer der gespeicherten Bewegungs- 
komponenten, die der bereitgestellten Spielin- to 
formation entspricht, auf der Grundlage der be- 
reitgestellten Spielinformation; und 
Erzeugen von Bilddaten, die eine Spielszene 
eines ausgewahlten Musikinstruments Oder 
Parts, die der Spielinformation entsprechen, 15 
synchron zu der bereitgestellten Spielinforma- 
tion und auf der Grundlage der ausgelesenen 
Bewegungskomponenten. 

Ton- und Bild-Erzeugungsverfahren nach Anspruch 20 
7, das ferner den Schritt des Erzeugens von ani- 
mierten Bilddaten aufweist, die der Spielinformation 
(MIDI) entsprechen, auf der Grundlage einer Infor- 
mation, die durch sequentielles Aneinanderfugen 
der ausgelesenen Bewegungskomponenten er- 25 
stellt wird, wobei die Vielzahl der Bewegungskom- 
ponenten eine Bewegungsbahn von Spielbewe- 
gungen eines subunterteilten Spielmusters fur je- 
des Musikinstrument oder jeden Spielpart repra- 
sentieren. 30 

Maschinenlesbares Aufzelchnungsmedium, das ei- 
ne Gruppe von Befehlen eines von einem Prozes- 
sor auszufuhrenden Ton- und Bild-Erzeugungsver- 
fahrens enthalt, wobei das Ton- und Bild-Erzeu- 
gungsverfahren die folgenden Schritte aufweist: 

Empfangen einer Spielinformation (MIDI); 
Erzeugen eines Tons auf der Grundlage der 
Spielinformation, die durch einen Empfangs- 
schritt empfangen wurde; 
Speichern einer Vielzahl von Bewegungskom- 
ponenten, die jeweils eine Beweguhgs informa- 
tion enthalten, die fur eine Bewegungsbahn 
((1), (2), (3), (4)) von Spielbewegungen eines 45 
. Spielmusters reprasentativ ist; 

Auslesen einer der gespeicherten Bewegungs- 
komponenten, die derempfangenen Spielinfor- 
mation entspricht, auf der Grundlage der bereit- 
gestellten Spielinformation; und so 
Erzeugen von Bilddaten, die eine Spielszene 
eines ausgewahlten Musikinstruments oder 
Parts, die der Spielinformation entsprechen, 
synchron zu der empfangenen Spielinformati- 
on und auf der Grundlage der ausgelesenen ss 
Bewegungskomponenten. 
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Revendications 

1. Dispositif de generation de sons et d'images 
comprenant : 

des moyens generateurs de sons (5) qui gene- 
rent un son base sur une information sur un 
6v6nement musical (MIDI); 
des moyens de memorisation (20) qui memori- 
sent une pluralite d'elements de mouvement 
comprenant chacun des informations de mou- 
vement representatives d'une trajectoire ((1), 
(2), (3), (4)) de mouvements d'execution d'un 
motif d'evenement musical ; et 
des moyens generateurs d'images qui, sur la 
base de rinformation sur un 6venement musi- 
cal, extraient desdits moyens de memorisation 
Tun des elements de mouvement qui corres- 
pond a rinformation sur rev6nement musical et, 
en synchronisation avec ladite information sur 
revenement musical et sur la base de I'element 
de mouvement extrait desdits moyens de me- 
morisation, g6nerent des donnees images qui 
illustrent une scene d'execution par un instru- 
ment musical ou une partie qui correspond k 
rinformation sur revenement musical. 

2. Dispositif de generation de sons et d'images selon 
la revendication 1, dans lequel. ledit element de 
mouvement represente une trajectoire de mouve- 
ments d'execution d'un motif d'evenement musical 
subdivise pourchaque instrument musical ou partie 
d'evenement musical, et dans lequel lesdits 
moyens generateurs d'images extraient desdits 
moyens de memorisation (20) I'un des elements de 
mouvement qui correspond a rinformation sur reve- 
nement musical et generent des donnees images 
animees qui correspondent h rinformation sur reve- 
nement musical sur la base de rinformation qui est 
creee en joignant sequentiellement les elements de 
mouvement extraits desdits moyens de memorisa- 
tion (20). 

3. Dispositif de generation de sons et d'images selon 
la revendication 1, dans lequel ledit element de 
mouvement comprend des informations de mouve- 
ment representatives d'une trajectoire ((1), (2), (3), 
(4)) de mouvements d'execution d'un motif d'eve- 
nement musical subdivise et un marqueur de points 
de sons qui indiquent les positions temporelles de 
generation de sons dans rinformation sur les mou- 
vements. 

1. Dispositif de generation de sons et d'images selon 
la revendication 1, dans lequel lesdites donnees 
images sont des donnees images animees tridi- 
mensionnelles. 
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5. Dispositif de generation de sons et d'images selon 
ta revendication 4, dans lequel un personnage qui 
represente un musicien et un point de vue illustres 
par les donnees images animees tridimensionnel- 

les peuvent etre changes par un operateur humain. s 

6. Dispositif de generation de sons et d'images selon 
la revendication 5, qui comprend de plus des 
moyens qui modifient Tinformation sur les mouve- 
ments en fonction d'un changement du personnage 10 
representant un musicien et du point de vue. 

7. Procede de generation de sons et d'images com- 
prenant les etapes qui consistent k : 

15 

fournir une information sur un evenement mu- 
sical (MIDI); 

generer un son sur la base de I'information four- 
hie sur un evenement musical ; 
memoriser une plurality d'elements de mouve- 20 
ment comprenant chacun des informations de 
mouvement representatives d'une trajectoire 
((1), (2), (3), (4)) de mouvements d'execution 
d'un motif d'evenement musical ; 
sur la base de I'information fournie sur un eve- 25 
nement musical, extraire I'un des elements de 
mouvement memorises qui correspond a I'in- 
formation fournie sur I'evenement musical ; et 
en synchronisation avec I'information fournie 
sur I'evenement musical, et sur la base de Tele- 30 
ment de mouvement extrait, generer des don- 
nees images qui illustrent une scene d'execu- 
tion par un instrument musical ou une partie qui 
correspond k I'information sur I'evenement mu- 
sical. 35 

8. Procede de generation de sons et d'images selon 
la revendication 7, qui comprend de plus I'etape 
consistant k generer des donnees images animees 

qui correspondent a ladite information sur un eve- 40 
nement musical (MIDI) sur la base d'une informa- 
tion qui est creee en joignant sequentiellement les 
elements de mouvement extraits, selon lequel ladi- 
te pluralite d'elements de mouvement represente 
une trajectoire de mouvements d'execution d'un 45 
motif d'evenement musical subdivise pour chaque 
instrument musical ou partie d'evenement musical. 

9. Support d'enregistrement lisible par une machine, 

qui contient un groupe d'instructions concemant un so 
procede de generation de sons et d'images k exe- 
cuter par un processeur, ledit procede de genera- 
tion de sons et d'images comprenant les etapes 
consistant k : 

55 

recevoir une information sur un evenement mu- 
sical (MIDI) ; 

generer u n son sur (a base de ladite information 



sur un evenement musical, re9ue pendant I'eta- 
pe de reception ; 

memoriser une pluralite d'elements de mouve- 
ment comprenant chacun des informations de 
mouvement representatives d'une trajectoire 
((1), (2), (3), (4)) de mouvements d'execution 
d'un motif d'evenement musical ; 
sur la base de I'information regue sur un eve- 
nement musical, extraire I'un des elements de 
mouvement memorises qui correspond k I'in- 
formation re$ue sur I'evenement musical ; et 
en synchronisation avec I'information reguesur 
I'evenement musical et sur la base de I'element 
de mouvement extrait, generer des donnees 
images qui illustrent une scene d'execution par 
un instrument musical ou une partie qui corres- 
pond k I'information regue sur I'evenement mu- 
sical. 
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